(1) Field of the Invention
The present invention relates to the field of systems for providing assistance in navigating a rotorcraft by processing images captured in flight and by constructing in flight and then dynamically displaying a representation of the outside world on the basis of the previously captured images.
The present invention relates more particularly to such navigation assistance systems that are suitable for displaying a dynamic representation of the outside world in a situation in which the pilot of a rotorcraft has lost visibility.
(2) Description of Related Art
Such a loss-of-visibility situation is caused in particular by the rotorcraft flying near the ground, which typically leads to clouds of particles being formed in the environment outside the rotorcraft. Said clouds of particles, such as dust or snow for example, give rise to a loss-of-visibility situation that is commonly referred to as “brown-out/white-out”.
Rotorcraft are aircraft in which lift is provided by at least one rotor having a substantially vertical axis, and they are suitable in particular for flying close to the ground not only at high speeds, but also typically at low speeds and/or while hovering. As an indication, a rotorcraft is commonly considered as having a low forward speed when its speed is less than 50 knots (kt) and high speeds of 125 kt or 150 kt or even more can be reached, in particular in a rotorcraft having propulsive propellers of substantially horizontal axis providing the rotorcraft with additional propulsion in translation.
Rotorcraft present the advantage of being able to fly under such flying conditions in any environment, which environment need not necessarily have been previously prepared or even identified.
Nevertheless, in this context, the problem of a brown-out/white-out situation arises when the rotorcraft is flying close to the ground and by way of indication at a height above the ground of less than 15 meters (m). Close to the ground, the rotor(s) of the rotorcraft raise clouds of particles that lead to a loss of visibility for the pilot of the rotorcraft. It is therefore useful to assist the pilot in navigation in the event of such a loss of visibility.
In order to provide such navigation assistance, it is known in a brown-out/white-out situation to provide the pilot with a display showing an artificial dynamic representation of the environment outside the rotorcraft, referred to below as the “outside world”. To this end, the representation of the outside world as displayed is constructed by a navigation assistance system on the basis of images that were captured in flight prior to the brown-out/white-out situation.
On this topic, reference may be made to the following documents EP 1 650 534 (EADS Deutschland), U.S. Pat. No. 7,642,929 (US Air Force), and U.S. Pat. No. 8,019,490 (Applied Minds), which disclose ways of implementing such navigation assistance systems in a brown-out/white-out situation.
Known solutions make use of at least one set of cameras facing towards the front of the rotorcraft, pointing together in the direction of a common front line of sight. The rotorcraft may also be fitted with at least one set of lateral cameras likewise pointing in the direction of a common lateral line of sight.
The concepts of “front”, “lateral”, “rear”, “right”, and “left” are commonly identified relative to the forward travel direction of the rotorcraft.
The cameras in a given set provide individual images of the environment outside the rotorcraft so as to provide the navigation assistance system with stereoscopic images giving terrain elevation data relating essentially to the topography of the terrain. The images, referred to below as “captured images”, are captured sequentially by the cameras at a given frequency in the form of digital data constituted by pixels.
Image processing means, such as texturing means for example, may optionally be used for processing the captured images by calculation in order to improve the visual effect of the display of the representation of the outside world derived from the images captured by the camera.
In the absence of a brown-out/white-out situation, and on request of the pilot, the current captured images picked up by the cameras may be displayed in order to provide the pilot with a dynamic display of a representation of the outside world, referred to below in this specific circumstance as the “current outside world”.
Furthermore, the on-board instrumentation of the rotorcraft includes in particular an inertial unit that conventionally delivers navigation data relating at least to the current state vector of the rotorcraft. The state vector of the rotorcraft represents in particular the relative or absolute position of the rotorcraft, the speed and more particularly the ground speed of the rotorcraft, and the orientation and/or the change in attitude of the rotorcraft as it progresses.
The navigation data of the rotorcraft is used by the navigation assistance system to construct a dynamic representation of the outside world that is displayed in a brown-out/white-out situation.
The current navigation data of the rotorcraft is incorporated in the images while they are being captured by generating metadata that is used by the navigation assistance system for constructing and displaying a representation of the outside world depending on how the current state vector of the rotorcraft varies, with this being referred to below in this specific circumstance as the “reconstituted world”.
For this purpose, simultaneous mapping and localizing means operate by calculation to generate maps with incorporated localization by performing known calculation processes, such as for example calculation processes commonly referred to as simultaneous localization and mapping (SLAM) or as concurrent mapping and localization (CML). By way of example, reference may be on this topic to the document “Visual SLAM for flying vehicles”, IEEE Transactions on Robotics, IEEE Service Center, Piscataway N.J., US, Vol. 24, No. 5, Oct. 1, 2008 (2008-10-01), pp. 1088-1093, DOI: 10.1109/TRO.2008.2004521.
A method of calculation using specific Kalman filters can be used to improve the pertinence of the maps that are constructed with incorporated localization. On this topic, reference may be made for example to the document “High resolution terrain mapping using low altitude aerial stereo imagery” (Il-Kyun Jung et al.), Proceedings of the ninth IEEE international conference on computer vision (ICCV), Nice, France, Oct. 13-16, 2003, International Conference on Computer Vision, Los Alamitos, Calif.: IEEE Comp. Soc, US, Vol. Conf. 9, Oct. 13, 2003, pp. 946-951, XP010662483, DOI: 10.11/ICCV 2003.1238450.
The simultaneous mapping and localization calculation processes are based on incrementally constructing maps with incorporated localization by means of predictive calculation algorithms conventionally making use of Kalman filters. The maps with incorporated localization are constructed from metadata incorporated in the captured images, which have allocated thereto the navigation data as identified sequentially by the on-board instrumentation simultaneously with the sequential camera pickup of the captured images.
The images captured individually at any given instant by the cameras in a given set are used to provide the navigation assistance system with metadata relating to the territories captured at a given instant by the cameras of a single set, referred to below as the “captured territories”. The maps with incorporated localization are thus successively generated in flight by a simultaneous mapping and localization calculation process and they are stored at a given repetition rate in a database for maps with incorporated localization.
In order to improve the pertinence of such databases of maps with incorporated localization, various sources of information may be used, such as those proposed in the document “Helicopter synthetic vision based DVE processing for all phases of flight” (Patrick O'Brien, David C. Baughman, H. Bruce Wallace) Degraded visual environments: enhanced, synthetic, and external vision solutions 2013, Vol. 8737, May 16, 2013 (2013-05-16), xp00273134, DOI: 10.1117:12.2016555.
The database of maps with incorporated localization is used in flight by a data processor unit to generate the construction and the dynamic display of the reconstituted world. For this purpose, the processor unit operates by calculation to compare the current navigation data of the rotorcraft with the navigation data incorporated in the various maps having incorporated localization that are stored in the database in order to generate the display of the reconstituted world.
More particularly, the processor unit identifies and extracts the maps with incorporated localization as a function of the variation in the current navigation data of the rotorcraft in order to construct and display the reconstituted world with display dynamics that vary depending on the current navigation data of the rotorcraft.
By way of example, reference may be made to Document EP 2 133 662 (Honeywell Int. Inc.), which discloses such ways of constructing and displaying a reconstituted world complying with the environment outside a host platform by means of a navigation assistance system that performs a process of simultaneous mapping and localization calculation process.
In this context, the reconstituted world may be displayed to provide the pilot with navigation assistance in the event of a sudden loss-of-visibility situation such as in the presence of fog banks or clouds of particles around the rotorcraft, in particular in a brown-out/white-out situation. For this purpose, the navigation assistance system incorporates means for detecting a brown-out/white-out situation that cause the processor unit to be put into operation and the reconstituted world to be displayed in order to mitigate the loss of the pilot's visibility by the environment outside the rotorcraft.
By way of example, such a brown-out/white-out situation may be detected:
by identifying the presence of a cloud of particles by analyzing the pixels that define the captured images, and more particularly by analyzing their density. The analysis of the pixels defining the captured images needs to be performed in ways that make it possible to identify the brown-out/white-out situation very quickly, while nevertheless being sufficiently reliable to authorize or not authorize the display of the reconstituted world depending on requirements;
by comparing by calculation the current height above the ground of the rotorcraft with a ground height threshold that conventionally gives rise to a brown-out/white-out situation, such as said distance between the rotorcraft and the ground being less than 15 m. Such a solution is commonly preferred since it makes it possible to obtain an identification of a brown-out/white-out situation rapidly on the basis of modest calculation capacity; and
more simply, by activation of a display control member by a pilot when confronted with a brown-out/white-out situation or, in a situation with visibility, by a pilot merely desiring to have the reconstituted world available in augmented reality with virtual images superposed on the real view.
In this context, the navigation assistance provided by constructing and displaying the reconstituted world requires considerable calculation power and memory. Such calculation power and memory must be available on board the rotorcraft in order to enable the reconstituted world to be constructed and displayed as quickly as possible depending on variation in the current navigation data of the rotorcraft with high-performance in the reliability, the visual quality, and the dynamic variation of the display.
More particularly, the calculation operations performed by the navigation assistance system require large calculation capacities and memory for achieving calculation frequencies that are good for:
capturing and processing images captured by the cameras as high-speed sequences in order to procure satisfactory visual quality for the captured images and in order to generate the metadata from which the maps with incorporated localization are prepared;
obtaining the largest possible number of accurate maps with incorporated localization that are available quickly in order to provide the processor unit with sufficient resources for constructing and displaying the reconstituted world and in order to provide the pilot with navigation assistance that is reliable and comfortable; and
displaying the reconstituted world with the displayed images being refreshed at high rates so as to cause the displayed reconstituted world to vary dynamically with display fluidity that is comfortable and reassuring for the pilot.
In addition, the reliability, the pertinence, and the fluidity with which the displayed dynamics of the reconstituted world vary depend on the immediate availability in good time of all of the information that needs to be mutually correlated during the various operations that lead to the reconstituted world being constructed and displayed.
As a result there is an ongoing search for a navigation assistance system of high performance that provides a display of the reconstituted world that is reliable, pertinent, and with good quality and good dynamic variation in the display, while nevertheless being subjected to a constraint involving limited capacity in terms of calculation power and memory needed for obtaining such a display quickly.
In the field of image processing by calculation, it is conventional to use processor units such as field programmable gate arrays (FPGAs). FPGA processor units or analogous processor units enable the logic circuits of the calculation means used by the processor units to be reconfigured. Depending on the choices made by the programmers concerning the ways in which the processor unit is to operate, the calculation logic circuits are configured to optimize data processing and exchanges between one another.
In the field of navigation assistance systems providing a dynamic display of the outside world, a common solution for optimizing the use of the capacities of the calculation and memory means on board a rotorcraft is to restrict the calculation operations that are to be performed by making use of a database of maps with incorporated localization that has been prepared on the ground prior to the flight mission of the rotorcraft. Such a database is constructed from images captured by any rotorcraft during previous flight missions.
Such a solution presents the advantage of reducing the calculation operations that need to be performed in flight and consequently of enabling the calculation and memory capacities of the navigation assistance system on board the rotorcraft to be reserved for constructing and displaying the reconstituted world.
Nevertheless, such a solution is not satisfactory because of the loss of pertinence in the information delivered by the database of maps with incorporated localization when they are prepared prior to the flight mission of the rotorcraft.
Furthermore, it is appropriate to optimize the use of an on-board navigation assistance system that provides, where necessary, a dynamic display of the representation of the outside world. For example, it is advantageous on the ground to make use of the information stored and/or generated during a flight mission by the navigation assistance system in order, a posteriori, to analyze said flight mission and/or the territories overflown by the rotorcraft.
Nevertheless, in addition to such potential uses, it would appear to be opportune to seek to implement other potential uses of the navigation assistance system. Such efforts should naturally be undertaken while taking account of constraints associated with searching for a high performance navigation assistance system while nevertheless having limited calculation and memory power but still capable of delivering in good time the information needed for constructing and displaying the reconstituted world in the context of the above-mentioned constraints.